Occupant state determining device, warning output control device, and occupant state determining method

ABSTRACT

An occupant state determining device includes an image data acquiring unit for acquiring image data indicating an image captured by a camera used for image capturing in a vehicle cabin, an image recognition processing unit for performing image recognition processing on the captured image by using the image data, a moveless state determining unit for determining whether an occupant is in a moveless state by using a result of the image recognition processing, and an abnormal state determining unit for determining whether the occupant is in an abnormal state due to decrease in the degree of awakening by using a result of the determination by the moveless state determining unit, and the abnormal state determining unit determines that the occupant is in an abnormal state when the duration of the moveless state exceeds a reference time.

TECHNICAL FIELD

The present disclosure relates to an occupant state determining device,a warning output control device, and an occupant state determiningmethod.

BACKGROUND ART

Conventionally, techniques of determining whether or not the driver isin an abnormal state due to decrease in the degree of awakening by usingan image captured by a camera for image capturing in a vehicle cabinhave been developed (refer to, for example, Patent Literatures 1 and 2).Hereafter, an abnormal state due to decrease in the degree of awakeningmay be simply referred to as an “abnormal state.”

The technique described in Patent Literature 1 is one of performingimage recognition processing on an image captured by a single camera,thereby detecting the eye opening degree of the driver. The techniquedescribed in Patent Literature 1 is one of determining whether or notthe driver is in an abnormal state on the basis of the detected eyeopening degree.

The technique described in Patent Literature 2 is one of determiningwhether or not the driver's head is included in an image captured byeach of multiple cameras, thereby detecting a change in the driver'sposture with respect to a forward inclination direction or a backwardinclination direction. The technique described in Patent Literature 2 isone of determining whether or not the driver is in an abnormal state onthe basis of the detected posture change.

CITATION LIST Patent Literature

Patent Literature 1: JP 2008-99884 A

Patent Literature 2: JP 2017-49636 A

SUMMARY OF INVENTION Technical Problem

Abnormal states due to decrease in the degree of awakening include astate in which a forward inclined posture occurs because of a loss ofconsciousness (referred to as a “forward inclined state” hereafter), astate in which a microsleep occurs with eyes open (referred to as a“microsleep state with eyes open” hereafter), a state in which amicrosleep occurs with eyes closed (referred to as a “microsleep statewith eyes closed” hereafter), and a state in which a backward inclinedposture occurs because of a loss of consciousness (referred to as a“backward inclined state” hereafter).

Out of these abnormal states, a microsleep state with eyes closed isaccompanied by a change in the eye opening degree, and a forwardinclined state and a backward inclined state are accompanied by aposture change. Therefore, a microsleep state with eyes closed can bedetermined by the technique described in Patent Literature 1, and aforward inclined state and a backward inclined state can be determinedby the technique described in Patent Literature 2. In contrast withthis, in a microsleep state with eyes open, a change in the eye openingdegree is small and a change in the posture is also small. Therefore, aproblem is that a microsleep state with eyes open cannot be determinedby the techniques described in Patent Literatures 1 and 2.

A further problem is that the technique described in Patent Literature 1cannot determine an abnormal state in a state in which the eye openingdegree is not detected normally (e.g., a state in which the driver iswearing sunglasses or the driver's eyes are hidden by the driver'sforelock).

The present disclosure is made in order to solve the above-mentionedproblems, and it is therefore an object of the present disclosure toprovide an occupant state determining device, a warning output controldevice, and an occupant state determining method capable of determiningan abnormal state including a microsleep state with eyes open, anddetermining an abnormal state regardless of whether or not an occupantis in a state in which the eye opening degree is not detected normally.

Solution to Problem

According to the present disclosure, an occupant state determiningdevice includes processing circuitry to acquire image data indicating animage captured by a camera used for image capturing in a vehicle cabin,perform image recognition processing on the captured image by using theimage data, determine whether an occupant is in a moveless state byusing a result of the image recognition processing, and determinewhether the occupant is in an abnormal state due to decrease in a degreeof awakening by using a result of the determination, wherein the imagerecognition processing includes a process of detecting a face area inthe captured image, when an amount of movement of the face area or anamount of change in a size of the face area is less than a referencequantity, the processing circuitry determines that the occupant is inthe moveless state, and the processing circuitry determines that theoccupant is in an abnormal state when a duration of the moveless stateexceeds a reference time.

Advantageous Effects of Invention

According to the present disclosure, because the configuration isprovided as above, an abnormal state including a microsleep state witheyes open can be determined, and an abnormal state can be determinedregardless of whether or not the occupant is in a state in which the eyeopening degree is not detected normally.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing a main part of a warning outputcontrol device according to Embodiment 1;

FIG. 2 is a block diagram showing a main part of an image recognitionprocessing unit included in the warning output control device accordingto Embodiment 1;

FIG. 3A is an explanatory drawing showing an example of a capturedimage, FIG. 3B is an explanatory drawing showing an example of a facearea, and FIG. 3C is an explanatory drawing showing an example ofmultiple feature points;

FIG. 4A is an explanatory drawing showing a positional relationshipamong multiple feature points which corresponds to a facial expressionof happiness, FIG. 4B is an explanatory drawing showing a positionalrelationship among multiple feature points which corresponds to a facialexpression of surprise, FIG. 4C is an explanatory drawing showing apositional relationship among multiple feature points which correspondsto a facial expression of fear, FIG. 4D is an explanatory drawingshowing a positional relationship among multiple feature points whichcorresponds to a facial expression of sadness, FIG. 4E is an explanatorydrawing showing a positional relationship among multiple feature pointswhich corresponds to a facial expression of anger, and FIG. 4F is anexplanatory drawing showing a positional relationship among multiplefeature points which corresponds to a facial expression of disgust;

FIG. 5 is an explanatory drawing showing an example of multiple featurepoints and a skeleton model of a driver;

FIG. 6A is a block diagram showing a hardware configuration of thewarning output control device according to Embodiment 1, and FIG. 6B isa block diagram showing another hardware configuration of the warningoutput control device according to Embodiment 1;

FIG. 7 is a flowchart showing the operation of the warning outputcontrol device according to Embodiment 1;

FIG. 8 is an explanatory drawing showing an example of a state in whicha face area in an n-th captured image and a face area in an (n+1)-thcaptured image overlap each other;

FIG. 9 is an explanatory drawing showing an example of PERCLOS;

FIG. 10A is an explanatory drawing showing an example of a predeterminedtime period associated with a reference histogram, FIG. 10B is anexplanatory drawing showing an example of the x and y coordinates of thecenter of a face area associated with the reference histogram, and FIG.10C is an explanatory drawing showing an example of the referencehistogram;

FIG. 11A is an explanatory drawing showing an example of a predeterminedtime period associated with a histogram for comparison, FIG. 11B is anexplanatory drawing showing an example of the x and y coordinates of thecenter of a face area associated with the histogram for comparison, andFIG. 11C is an explanatory drawing showing an example of the histogramfor comparison;

FIG. 12 is a block diagram showing a main part of a warning outputcontrol device according to Embodiment 2; and

FIG. 13 is a flowchart showing the operation of the warning outputcontrol device according to Embodiment 2.

DESCRIPTION OF EMBODIMENTS

Hereafter, in order to explain the present disclosure in greater detail,embodiments of the present disclosure will be described with referenceto the accompanying drawings.

Embodiment 1

FIG. 1 is a block diagram showing a main part of a warning outputcontrol device according to Embodiment 1. FIG. 2 is a block diagramshowing a main part of an image recognition processing unit included inthe warning output control device according to Embodiment 1. An occupantstate determining device 100 of Embodiment 1 and the warning outputcontrol device 200 will be explained by reference to FIGS. 1 and 2.

A vehicle 1 has a camera 2 used for image capturing in the cabinthereof. The camera 2 is disposed in, for example, the center cluster ofthe vehicle 1. The camera 2 includes, for example, an infrared camera ora visible light camera.

An image data acquiring unit 11 acquires image data showing an image Icaptured by the camera 2 from the camera 2 at predetermined timeintervals. The image data acquiring unit 11 outputs the acquired imagedata to an image recognition processing unit 12.

The image recognition processing unit 12 performs image recognitionprocessing on the captured image I by using the image data outputted bythe image data acquiring unit 11. The image recognition processing bythe image recognition processing unit 12 includes, for example,below-mentioned processes by a face area detecting unit 21, a facefeature point detecting unit 22, an eye opening degree detecting unit23, an inclination angle detecting unit 24, a facial expressiondetecting unit 25, a human body area detecting unit 31, a skeletal framefeature point detecting unit 32, and an inclination angle detecting unit33.

The face area detecting unit 21 detects an area (referred to as a “facearea” hereafter) A1 in the captured image I, the area corresponding tothe face of the driver of the vehicle 1. As a method of detecting theface area A1, well-known various methods can be used, and a detailedexplanation of the method will be omitted hereafter. Hereafter, thedriver of the vehicle 1 is simply referred to as the “driver.”

The face feature point detecting unit 22 detects multiple feature pointsP1 in the face area A1, the multiple feature points P1 corresponding toseveral face parts of the driver. As a method of detecting featurepoints P1, well-known various methods can be used, and a detailedexplanation of the method will be omitted hereafter.

Concretely, for example, the face feature point detecting unit 22detects multiple feature points P1 corresponding to the driver's righteye, multiple feature points P1 corresponding to the driver's left eye,multiple feature points P1 corresponding to the driver's nose, andmultiple feature points P1 corresponding to the driver's mouth. Themultiple feature points P1 corresponding to the driver's right eyeinclude, for example, a feature point P1 corresponding to the outer eyecorner, a feature point P1 corresponding to the inner eye corner, afeature point P1 corresponding to the upper eyelid, and a feature pointP1 corresponding to the lower eyelid. The multiple feature points P1corresponding to the driver's left eye include, for example, a featurepoint P1 corresponding to the outer eye corner, a feature point P1corresponding to the inner eye corner, a feature point P1 correspondingto the upper eyelid, and a feature point P1 corresponding to the lowereyelid. The multiple feature points P1 corresponding to the driver'snose include, for example, a feature point P1 corresponding to the nasalroot, a feature point P1 corresponding to the nasal bridge, featurepoints P1 corresponding to the nasal wings, and a feature point P1corresponding to the nasal tip. The multiple feature points P1corresponding to the driver's mouth include, for example, a featurepoint P1 corresponding to the upper lip and a feature point P1corresponding to the lower lip.

FIG. 3A shows an example of the captured image I, FIG. 3B shows anexample of the face area A1, and FIG. 3C shows an example of themultiple feature points P1. In the example shown in FIG. 3C, fourfeature points P1 corresponding to the driver's right eye, four featurepoints P1 corresponding to the driver's left eye, two feature points P1corresponding to the driver's nose, and three feature points P1corresponding to the driver's mouth are detected.

The eye opening degree detecting unit 23 detects the eye opening degreeD of the driver by using multiple feature points P1 detected by the facefeature point detecting unit 22 (more concretely, the multiple featurepoints P1 corresponding to the driver's right eye and the multiplefeature points P1 corresponding to the driver's left eye). As a methodof detecting the eye opening degree D, well-known various methods can beused, and the method is not limited to the following concrete example.

Concretely, for example, the eye opening degree detecting unit 23calculates a current value of the distance between the feature point P1corresponding to the upper eyelid and the feature point P1 correspondingto the lower eyelid (referred to as the “distance between the eyelids”hereafter). The eye opening degree detecting unit 23 calculates areference value of the distance between the eyelids on the basis of themode of the distances between the eyelids which are acquired within apredetermined time period. The eye opening degree detecting unit 23calculates the eye opening degree D on the basis of the ratio of thecurrent value to the reference value. More specifically, the unit of theeye opening degree D is percent.

The inclination angle detecting unit 24 detects inclination angles θ1 ofthe driver's head by using multiple feature points P1 detected by theface feature point detecting unit 22. As a method of detecting aninclination angle θ1, well-known various methods can be used, and adetailed explanation of the method will be omitted.

Inclination angles θ1 are detected with respect to, for example, a statein which the driver's face is directed toward the front of the vehicle1, and the back of the driver's head is in contact with the headrest ofthe driver's seat (i.e., 0 degrees). For example, the inclination angledetecting unit 24 detects an inclination angle θ1 in a rotationdirection around the driver's neck, an inclination angle θ1 with respectto a forward inclination direction or a backward inclination direction,and an inclination angle θ1 with respect to a rightward or leftwarddirection.

The facial expression detecting unit 25 detects the driver's facialexpression by using multiple feature points P1 detected by the facefeature point detecting unit 22. As a method of detecting a facialexpression, well-known various methods can be used, and the method isnot limited to the following concrete example.

Concretely, for example, the facial expression detecting unit 25 detectsthe driver's facial expression on the basis of a positional relationshipamong three feature points P1 corresponding to the driver's righteyebrow, three feature points P1 corresponding to the driver's lefteyebrow, two feature points P1 corresponding to the driver's right eye,two feature points P1 corresponding to the driver's left eye, and fourfeature points P1 corresponding to the driver's mouth, i.e., apositional relationship among the 14 feature points P1 in total. FIG. 4Ashows an example of the positional relationship among the 14 featurepoints P1 which corresponds to a facial expression of happiness. FIG. 4Bshows an example of the positional relationship among the 14 featurepoints P1 which corresponds to a facial expression of surprise. FIG. 4Cshows an example of the positional relationship among the 14 featurepoints P1 which corresponds to a facial expression of fear. FIG. 4Dshows an example of the positional relationship among the 14 featurepoints P1 which corresponds to a facial expression of sadness. FIG. 4Eshows an example of the positional relationship among the 14 featurepoints P1 which corresponds to a facial expression of anger. FIG. 4Fshows an example of the positional relationship among the 14 featurepoints P1 which corresponds to a facial expression of disgust.

The human body area detecting unit 31 detects an area A2 correspondingto the driver's face and body in the captured image I (referred to as a“human body area” hereafter). As a method of detecting the human bodyarea A2, well-known various methods can be used, and a detailedexplanation of the method will be omitted.

The skeletal frame feature point detecting unit 32 detects multiplefeature points P2 in the human body area A2 which are used for thegeneration of a so-called “skeleton model.” As a method of detecting thefeature points P2, well-known various methods can be used, and adetailed explanation of the method will be omitted.

The inclination angle detecting unit 33 generates a skeleton model forthe driver by using the multiple feature points P2 detected by theskeletal frame feature point detecting unit 32. The inclination angledetecting unit 33 detects inclination angles θ2 of the driver'sshoulders, inclination angles θ3 of the driver's arms, and inclinationangles θ4 of the driver's head by using the generated skeleton model. Asa method of detecting the inclination angles θ2, θ3, and θ4, well-knownvarious methods can be used, and a detailed explanation of the methodwill be omitted.

The inclination angles θ2, θ3, and θ4 are detected with respect to, forexample, a state in which the driver's face is directed toward the frontof the vehicle 1, the back of the driver's head is in contact with theheadrest of the driver's seat, the driver's back is in contact with thebackrest of the driver's seat, the driver is holding the steering wheelof the vehicle 1 with both hands, and the steering angle of the vehicle1 is approximately 0 degrees (in other words, the vehicle 1 issubstantially going straight). Because a concrete example of theinclination angles θ4 is the same as that of the inclination angles θ1,a detailed explanation of the concrete example will be omitted.

FIG. 5 shows an example of the multiple feature points P2 and theskeleton model for the driver. In the example shown in FIG. 5, 12feature points P2 are detected and a skeleton model based on a so-called“stick model” is generated.

A moveless state determining unit 13 determines whether or not thedriver is in a state in which there is no movement of the driver(referred to as a “moveless state” hereafter) by using a result of theimage recognition processing by the image recognition processing unit12. Concrete examples of a determining method used by the moveless statedetermining unit 13 will be mentioned later.

An abnormal state determining unit 14 determines whether or not thedriver is in an abnormal state due to decrease in the degree ofawakening by using a result of the determination by the moveless statedetermining unit 13. Hereafter, an abnormal state due to decrease in thedegree of awakening is simply referred to as an “abnormal state.”

More specifically, the abnormal state determining unit 14 calculates theduration T of the moveless state by using the determination resultprovided by the moveless state determining unit 13. The abnormal statedetermining unit 14 compares the calculated duration T with apredetermined reference time Tth. When the duration T exceeds thereference time Tth, the abnormal state determining unit 14 determinesthat the driver is in an abnormal state.

When the abnormal state determining unit 14 determines that the driveris in an abnormal state, a warning output control unit 15 performscontrol to output a warning.

Concretely, for example, the warning output control unit 15 performscontrol to cause a display device 3 to display an image for warning. Asan alternative, for example, the warning output control unit 15 performscontrol to cause a sound output device 4 to output a sound for warning.More specifically, the warning using an image or a sound is intendedmainly for the inside of the vehicle.

As an alternative, for example, the warning output control unit 15performs control to cause a wireless communication device 5 to transmita signal for warning. The signal is transmitted to a so-called “center”via a wide area network like the Internet. As an alternative, the signalis transmitted to other vehicles traveling in the vicinity of thevehicle 1 via “vehicle-to-vehicle communications.” More specifically,the warning using a signal is intended mainly for the outside of thevehicle.

The display device 3 includes, for example, a liquid crystal display(LCD), an organic electro-luminescence display (OLED), or a head-updisplay (HUD). The sound output device 4 includes, for example, aspeaker. The wireless communication device 5 includes, for example, atransmitter for and a receiver for connection with the Internet or atransmitter for and a receiver for vehicle-to-vehicle communications.

The warning output control unit 15 may perform any two or more of thecontrol to cause the display device 3 to display an image for warning,the control to cause the sound output device 4 to output a sound forwarning, and the control to cause the wireless communication device 5 totransmit a signal for warning.

A main part of the image recognition processing unit 12 is constitutedby the face area detecting unit 21, the face feature point detectingunit 22, the eye opening degree detecting unit 23, the inclination angledetecting unit 24, the facial expression detecting unit 25, the humanbody area detecting unit 31, the skeletal frame feature point detectingunit 32, and the inclination angle detecting unit 33. A main part of theoccupant state determining device 100 is constituted by the image dataacquiring unit 11, the image recognition processing unit 12, themoveless state determining unit 13, and the abnormal state determiningunit 14. A main part of the warning output control device 200 isconstituted by the occupant state determining device 100 and the warningoutput control unit 15.

Next, the hardware configuration of the main part of the warning outputcontrol device 200 will be explained by reference to FIG. 6.

As shown in FIG. 6A, the warning output control device 200 isconstituted by a computer, and the computer has a processor 41 and amemory 42. In the memory 42, a program for causing the computer tofunction as the image data acquiring unit 11, the image recognitionprocessing unit 12, the moveless state determining unit 13, the abnormalstate determining unit 14, and the warning output control unit 15 isstored. The functions of the image data acquiring unit 11, the imagerecognition processing unit 12, the moveless state determining unit 13,the abnormal state determining unit 14, and the warning output controlunit 15 are implemented by the processor 41's reading and executing theprogram stored in the memory 42.

As an alternative, as shown in FIG. 6B, the warning output controldevice 200 may have a processing circuit 43. In this case, the functionsof the image data acquiring unit 11, the image recognition processingunit 12, the moveless state determining unit 13, the abnormal statedetermining unit 14, and the warning output control unit 15 may beimplemented by the processing circuit 43.

As an alternative, the warning output control device 200 may have theprocessor 41, the memory 42, and the processing circuit 43. In thiscase, a part of the functions of the image data acquiring unit 11, theimage recognition processing unit 12, the moveless state determiningunit 13, the abnormal state determining unit 14, and the warning outputcontrol unit 15 may be implemented by the processor 41 and the memory42, and the remaining part of the functions may be implemented by theprocessing circuit 43.

As the processor 41, for example, a central processing unit (CPU), agraphics processing unit (GPU), a microprocessor, a microcontroller, ora digital signal processor (DSP) is used.

As the memory 42, for example, a semiconductor memory, a magnetic disc,an optical disc, or a magneto optical disc is used. More concretely, asthe memory 42, a random access memory (RAM), a read only memory (ROM), aflash memory, an erasable programmable read only memory (EPROM), anelectrically erasable programmable read-only memory (EEPROM), a solidstate drive (SSD), a hard disk drive (HDD), a floppy disk (FD), acompact disc (CD), a digital versatile disc (DVD), a magneto-optical(MO), a mini disc (MD), or the like is used.

As the processing circuit 43, for example, an application specificintegrated circuit (ASIC), a programmable logic device (PLD), afield-programmable gate array (FPGA), a system-on-a-chip (SoC), or asystem large-scale integration (LSI) is used.

Next, the operation of the warning output control device 200 will beexplained by reference to a flowchart of FIG. 7. The warning outputcontrol device 200 repeatedly performs the processing shown in FIG. 7 ina state in which, for example, the power is switched on (moreconcretely, in a state in which the ignition power source of the vehicle1 is switched on).

First, in step ST1, the image data acquiring unit 11 acquires the imagedata indicating the captured image I from the camera 2. The image dataacquiring unit 11 outputs the acquired image data to the imagerecognition processing unit 12.

Then, in step ST2, the image recognition processing unit 12 performs theimage recognition processing on the captured image I by using the imagedata outputted by the image data acquiring unit 11. Because a concreteexample of the image recognition processing is already explained, anexplanation of the concrete example will not be repeated.

Then, in step ST3, the moveless state determining unit 13 determineswhether or not the driver is in the moveless state by using a result ofthe image recognition processing by the image recognition processingunit 12. Concrete examples of the determining method used by themoveless state determining unit 13 will be mentioned later.

When it is determined that the driver is in the moveless state (“YES” instep ST3), the abnormal state determining unit 14, in step ST4,determines whether the duration T of the moveless state exceeds thereference time Tth. This duration T is calculated by the abnormal statedetermining unit 14.

When it is determined that the duration T of the moveless state exceedsthe reference time Tth (“YES” in step ST4), the warning output controlunit 15, in step ST5, performs the control to output a warning. Becausea concrete example of a method of outputting a warning, the method beingused by the warning output control unit 15, is already explained, anexplanation of the concrete example will not be repeated.

Next, concrete examples of the determining method used by the movelessstate determining unit 13 will be explained by reference to FIGS. 8 to11.

First Concrete Example of the Determining Method Used by the MovelessState Determining Unit 13

The moveless state determining unit 13 calculates, as to the face areasA1 _(n) and A1 _(n+1) in the captured images I_(n) and I_(n+1) that aretwo consecutive frames in time (n is an arbitrary integer), the area ofa part in which the face areas A1 _(n) and A1 _(n+1) overlap each other(referred to as the “overlapping area” hereafter) by using a result ofthe detection by the face area detecting unit 21. The moveless statedetermining unit 13 calculates the ratio of the overlapping area to thearea of the face area A1 _(n) or A1 _(n+1) (referred to as the“overlapping area ratio” hereafter). The moveless state determining unit13 calculates the amount of movement of the face area A1 on the basis ofthe overlapping area ratio. For example, the moveless state determiningunit 13 calculates the amount of movement in a way that the amount ofmovement of the face area A1 increases as the overlapping area ratiodecreases. When the amount of movement of the face area A1 is less thana predetermined reference quantity, the moveless state determining unit13 determines that the driver is in the moveless state. The denominatorof the overlapping area ratio may be the area of a part corresponding tothe sum (more concretely, the logical sum) of the face areas A1 _(n) andA1 _(n+1).

FIG. 8 shows an example of a state in which the face area A1 _(n) in then-th captured image I_(n) and the face area A1 _(n+1) in the (n+1)-thcaptured image I_(n+1) overlap each other. In the figure, a hatched areashows the part in which the face areas A1 _(n) and A1 _(n+1) overlapeach other. As mentioned above, the captured images I_(n) and I_(n+1)correspond to two consecutive frames in time. More specifically, theimages I_(n) and I_(n+1) are captured at mutually different times.

Second Concrete Example of the Determining Method Used by the MovelessState Determining Unit 13

The moveless state determining unit 13 calculates, as to the face areasA1 _(n) and A1 _(n+1) in the captured images I_(n) and I_(n+1) that aretwo consecutive frames in time (n is an arbitrary integer), the amountof change in the size of the face area A1 _(n+1) with respect to thesize of the face area A1 _(n) by using the detection result provided bythe face area detecting unit 21. This calculated amount of change showsthe amount of movement of the head of the driver in a forward orbackward direction. When the amount of change in the size of the facearea A1 is less than a predetermined reference quantity, the movelessstate determining unit 13 determines that the driver is in the movelessstate.

Third Concrete Example of the Determining Method Used by the MovelessState Determining Unit 13

The moveless state determining unit 13 calculates the amount of movementof the center of the face area A1 in each of the images I that arecaptured for the second and subsequent times within a predetermined timeperiod, with respect to the center of the face area A1 in the capturedimage I that is captured for the first time within the predeterminedtime period, by using the detection result provided by the face areadetecting unit 21. The moveless state determining unit 13 calculates theamount of movement of the face area A1 within the predetermined timeperiod by adding up the calculated amounts of movement. When the amountof movement of the face area A1 is less than a predetermined referencequantity, the moveless state determining unit 13 determines that thedriver is in the moveless state.

Fourth Concrete Example of the Determining Method Used by the MovelessState Determining Unit 13

The moveless state determining unit 13 determines whether or not aninclination angle θ1 of the driver's head is equal to or greater than apredetermined reference angle by using a result of the detection by theinclination angle detecting unit 24. The moveless state determining unit13 determines whether or not the amount of movement of the face area A1is less than a reference quantity by using the same method as that inthe first or third concrete example. When the inclination angle θ1 isequal to or greater than the reference angle and the amount of movementof the face area A1 is less than the reference quantity, the movelessstate determining unit 13 determines that the driver is in the movelessstate.

Fifth Concrete Example of the Determining Method Used by the MovelessState Determining Unit 13

The moveless state determining unit 13 determines whether or not aninclination angle θ1 of the driver's head is equal to or greater than apredetermined reference angle by using the detection result provided bythe inclination angle detecting unit 24. The moveless state determiningunit 13 determines whether or not the amount of change in the size ofthe face area A1 is less than a reference quantity by using the samemethod as that in the second concrete example. When the inclinationangle θ1 is equal to or greater than the reference angle and the amountof change in the size of the face area A1 is less than the referencequantity, the moveless state determining unit 13 determines that thedriver is in the moveless state.

Sixth Concrete Example of the Determining Method Used by the MovelessState Determining Unit 13

In general, human eyeblinks are classified into three types. Morespecifically, they are a “periodic eyeblink” that is performedunconsciously, a “reflexive eyeblink” that is performed when light comesinto eyes, and a “spontaneous eyeblink” that is performed spontaneously.In a state in which a human being is awake, a periodic eyeblink isperformed at approximately fixed time intervals (i.e., with anapproximately fixed frequency).

Accordingly, the moveless state determining unit 13 calculates thefrequency of the driver's eyeblinks by using a result of the detectionby the eye opening degree detecting unit 23. When the calculatedfrequency is less than a predetermined threshold, the moveless statedetermining unit 13 determines that the driver is in the moveless state.

Seventh Concrete Example of the Determining Method Used by the MovelessState Determining Unit 13

The moveless state determining unit 13 calculates so-called “PERCLOS” byusing the detection result provided by the eye opening degree detectingunit 23. PERCLOS shows the ratio of time periods T_(CLOSE)(s) duringeach of which the driver's eyes are closed to a predetermined timeperiod (a so-called “window size”) T_(WINDOW). Therefore, the followingequation (1) is used for the calculation of PERCLOS.PERCLOS=Σ(T _(CLOSE)(s))/T _(WINDOW)  (1)

FIG. 9 shows an example of PERCLOS. In the figure, Dth is a thresholdthat is an object to be compared with the eye opening degree D, and isused for the determination of whether the driver is either in a state inwhich the driver's eyes are open or in a state in which the driver'seyes are closed. The moveless state determining unit 13 calculates, forexample, each of T_(CLOSE)(s) by using the predetermined threshold Dthand calculates Σ(T_(CLOSE)(s)) that is the sum total of theseT_(CLOSE)(s), thereby calculating PERCLOS.

The moveless state determining unit 13 calculates PERCLOS for eachpredetermined time period (i.e., each T_(WINDOW)), and calculates theamount of change in these PERCLOS(s). When the amount of change inPERCLOS(s) is less than a predetermined reference quantity, the movelessstate determining unit 13 determines that the driver is in the movelessstate.

Eighth Concrete Example of the Determining Method Used by the MovelessState Determining Unit 13

The moveless state determining unit 13 acquires information showing theposition coordinates of the centers of the face areas A1 ₀ to A1 _(n−1)in the captured images I₀ to I_(n−1) of n frames (more concretely, the xcoordinates x₀ to x_(n−1) and the y coordinates y₀ to y_(n−1)) from theface area detecting unit 21 (n is an integer equal to or greater than2). The images I₀ to I_(n−1) are captured within a predetermined timeperiod (e.g., A+α seconds) starting from a time t1 when a predeterminedcondition (referred to as a “calibration condition” hereafter) isestablished. When the image capturing frame rate of the camera 2 isexpressed by frame_rate, the relation between n and A is shown by thefollowing equation (2).n=A*frame_rate  (2)

FIG. 10A shows an example of the predetermined time period (i.e., A+αseconds). FIG. 10B shows an example of the x coordinates x₀ to x_(n−1)and the y coordinates y₀ to y_(n−1) of the centers of the face areas A1₀ to A1 _(n−1).

The moveless state determining unit 13 sets up multiple bins inone-to-one correspondence with multiple areas which are spaced atpredetermined intervals (e.g., at intervals of B pixels) and into whicheither the whole of each captured image I or a part of each capturedimage I (e.g., a part having a greater probability of including the facearea A1 in the captured image I than other parts) is divided. Themoveless state determining unit 13 generates a histogram in which, as tothe n face areas A1 ₀ to A1 _(n−1), the number of centers of face areasA1, the centers being included in each bin, is counted. Hereafter, thishistogram is referred to as the “reference histogram.” FIG. 10C shows anexample of the reference histogram.

After that, the moveless state determining unit 13 acquires informationindicating the position coordinates of the centers of the face areas A1₀ to A1 _(m−1) in the captured images I₀ to I_(m−1) of m frames (moreconcretely, the x coordinates x₀ to x_(m−1) and the y coordinates y₀ toy_(m−1)) from the face area detecting unit 21 (m is an integer equal toor greater than 2). The images I₀ to I_(m−1) are captured within apredetermined time period (e.g., C seconds) ending at a time t2 when theimage I_(m−1) is captured. When the image capturing frame rate of thecamera 2 is expressed by frame_rate, the relation between m and C isshown by the following equation (3).m=C*frame_rate  (3)

FIG. 11A shows an example of the predetermined time period (i.e., Cseconds). FIG. 11B shows an example of the x coordinates x₀ to x_(m−1)and the y coordinates y₀ to y_(m−1) of the centers of the face areas A1₀ to A1 _(m−1).

The moveless state determining unit 13 sets up multiple bins which aresimilar to those for the reference histogram. The moveless statedetermining unit 13 generates a histogram in which, as to the m faceareas A1 ₀ to A1 _(m−1), the number of centers of face areas A1, thecenters being included in each bin, is counted. Hereafter, thishistogram is referred to as the “histogram for comparison.” FIG. 11Cshows an example of the histogram for comparison.

The moveless state determining unit 13 compares the value of each bin inthe histogram for comparison with the value of the corresponding bin inthe reference histogram. As a result, the moveless state determiningunit 13 determines the degree of change in the histogram for comparisonwith respect to the reference histogram. Concretely, for example, themoveless state determining unit 13 determines either the degree ofchange in the distribution of values in the histogram or presence orabsence of change in the position of a bin corresponding to a maximum.As a result, the moveless state determining unit 13 determines presenceor absence of change in the driver's face. When it is determined thatthere is no change in the driver's face, the moveless state determiningunit 13 determines that the driver is in the moveless state.

The histogram for comparison may be generated repeatedly atpredetermined time intervals after the reference histogram is generated(more specifically, the histogram for comparison may be updated at thepredetermined time intervals). Every time the histogram for comparisonis generated (more specifically, every time the histogram for comparisonis updated), the moveless state determining unit 13 may compare thevalue of each bin in the newest histogram for comparison with the valueof the corresponding bin in the reference histogram.

Ninth Concrete Example of the Determining Method Used by the MovelessState Determining Unit 13

Generally, in a state in which a human being is awake, his or her facialexpression changes at approximately fixed time intervals (i.e., with anapproximately fixed frequency) depending on the ambient environment, thecontent of conversation, or the like. In contrast with this, no changein the facial expression occurs in a state in which the degree ofawakening decreases. More concretely, a facial expression in which thecorners of the mouth are lowered is maintained because of relaxation ofmimic muscles.

Accordingly, the moveless state determining unit 13 determines presenceor absence of change in the driver's facial expression by using a resultof the detection by the facial expression detecting unit 25. Whendetermining that there is no change in the facial expression, themoveless state determining unit 13 determines that the driver is in themoveless state.

Tenth Concrete Example of the Determining Method Used by the MovelessState Determining Unit 13

The moveless state determining unit 13 determines presence or absence ofa change in the inclination angles θ2 (i.e., presence or absence of amovement of the driver's shoulder) by using a result of the detection bythe inclination angle detecting unit 33. Further, the moveless statedetermining unit 13 determines presence or absence of a change in theinclination angles θ3 (i.e., presence or absence of a movement of thedriver's arm). Further, the moveless state determining unit 13determines presence or absence of a change in the inclination angles θ4(i.e., presence or absence of a movement of the driver's head).

The moveless state determining unit 13 determines whether or not thedriver is in the moveless state on the basis of results of thedetermination of presence or absence of these movements. For example,when there are no movements of the driver's shoulders, there are nomovements of the driver's arms, and there is no movement of the driver'shead, the moveless state determining unit 13 determines that the driveris in the moveless state.

The moveless state determining unit 13 may execute any one of the firstto tenth concrete examples or any two or more of the first to tenthconcrete examples. In the case of executing any two or more of the firstto tenth concrete examples, when it is determined, by using apredetermined number of methods out of the two or more methods, that thedriver is in the moveless state, the moveless state determining unit 13may output a determination result indicating that the driver is in themoveless state to the abnormal state determining unit 14. As analternative, in this case, the moveless state determining unit 13 mayperform weighting on determination results provided by the methods,thereby outputting a final determination result to the abnormal statedetermining unit 14.

As mentioned above, the occupant state determining device 100 ofEmbodiment 1 determines whether or not the driver is in the movelessstate and determines whether or not the driver is in an abnormal stateon the basis of the duration T of the moveless state. Further, themethods of determining whether or not the driver is in the movelessstate include methods not using a result of the detection of the eyeopening degree D (the first, second, third, fourth, fifth, eighth,ninth, and tenth concrete examples). Therefore, an abnormal stateincluding a microsleep state with eyes open can be determined, and anabnormal state can be determined regardless of whether or not the driveris in a state in which the eye opening degree D is not detectednormally.

Further, any of the methods of determining whether or not the driver isin the moveless state (the first to tenth concrete examples) uses theimage I captured by the camera 2. More specifically, expensive devicessuch as living body sensors, are unnecessary. Therefore, the whole of asystem including the warning output control device 200 can beimplemented at a low cost.

The abnormal state determining unit 14 may acquire informationindicating a so-called “autonomous driving level” from an electroniccontrol unit (ECU) for autonomous driving control disposed in thevehicle 1. The autonomous driving level is shown by a value ranging from0 to 5, and the level 0 shows that the vehicle 1 is traveling whilebeing driven manually. The abnormal state determining unit 14 may setthe reference time Tth to a value that differs depending on theautonomous driving level of the vehicle 1 by using the acquiredinformation. For example, when the autonomous driving level is 2 orless, the abnormal state determining unit 14 may set the reference timeTth to a value smaller compared with that in the case in which theautonomous driving level is 3 or more. As a result, the determination ofan abnormal state on the basis of a proper reference time Tth can beimplemented in accordance with the autonomous driving level of thevehicle 1.

Further, when performing the process of detecting the eye opening degreeD, the eye opening degree detecting unit 23 may output informationindicating the success or failure of the detection to the moveless statedetermining unit 13. By using the information outputted by the eyeopening degree detecting unit 23, when the eye opening degree detectingunit 23 has succeeded in the detection of the eye opening degree D, themoveless state determining unit 13 may determine whether or not thedriver is in the moveless state by means of a method using a result ofthe detection of the eye opening degree D (at least one of the sixth andseventh concrete examples), whereas when the eye opening degreedetecting unit 23 has failed in the detection of the eye opening degreeD, the moveless state determining unit 13 may determine whether or notthe driver is in the moveless state by means of a method not using aresult of the detection of the eye opening degree D (at least one of thefirst, second, third, fourth, fifth, eighth, ninth, and tenth concreteexamples). As a result, the determination of the moveless state using aproper method can be implemented in accordance with the success orfailure of the detection of the eye opening degree D.

Further, when having succeeded in the detection of the eye openingdegree D, the eye opening degree detecting unit 23 may outputinformation indicating the reliability of the result of the detection tothe abnormal state determining unit 14. The abnormal state determiningunit 14 may set the reference time Tth to a value that differs dependingon the reliability of the result of the detection of the eye openingdegree D by using the information outputted by the eye opening degreedetecting unit 23. For example, when the reliability is less than apredetermined threshold, the abnormal state determining unit 14 may setthe reference time Tth to a value smaller compared with that in the casein which the reliability is equal to or greater than the threshold. As aresult, the determination of an abnormal state on the basis of a properreference time Tth can be implemented in accordance with the reliabilityof the result of the detection of the eye opening degree D.

Further, in a case in which the determining methods used by the movelessstate determining unit 13 do not include the fourth and fifth concreteexamples, the image recognition processing unit 12 may be configured soas to exclude the inclination angle detecting unit 24 shown in FIG. 2.

Further, in a case in which the determining methods used by the movelessstate determining unit 13 do not include the sixth and seventh concreteexamples, the image recognition processing unit 12 may be configured soas to exclude the eye opening degree detecting unit 23 shown in FIG. 2.

Further, in a case in which the determining methods used by the movelessstate determining unit 13 do not include the ninth concrete example, theimage recognition processing unit 12 may be configured so as to excludethe facial expression detecting unit 25 shown in FIG. 2.

Further, in a case in which the determining methods used by the movelessstate determining unit 13 do not include the tenth concrete example, theimage recognition processing unit 12 may be configured so as to excludethe human body area detecting unit 31, the skeletal frame feature pointdetecting unit 32, and the inclination angle detecting unit 33 which areshown in FIG. 2.

Further, the warning outputting method used by the warning outputcontrol unit 15 is not limited to the above-mentioned concrete example.For example, the warning output control unit 15 may perform control tolight the hazard lamp of the vehicle 1 when the abnormal statedetermining unit 14 determines that the driver is in an abnormal state.

Further, the occupant state determining device 100 can be used not onlyfor the determination of whether or not the driver is in an abnormalstate, but also for the determination of whether or not an occupantdifferent from the driver, out of the occupants in the vehicle 1, is inan abnormal state. For example, when the vehicle 1 is traveling based onautonomous driving with level 3 or higher, the occupant statedetermining device 100 can also be used for the determination of whetheror not an occupant sitting in the driver's seat, but not driving thevehicle 1 is in an abnormal state.

As mentioned above, the occupant state determining device 100 ofEmbodiment 1 includes the image data acquiring unit 11 that acquiresimage data indicating an image I captured by the camera 2 used for imagecapturing in the vehicle cabin, the image recognition processing unit 12that performs the image recognition processing on the captured image Iby using the image data, the moveless state determining unit 13 thatdetermines whether or not an occupant is in the moveless state by usinga result of the image recognition processing, and the abnormal statedetermining unit 14 that determines whether or not the occupant is in anabnormal state due to decrease in the degree of awakening by using aresult of the determination by the moveless state determining unit 13,and the abnormal state determining unit 14 determines that the occupantis in an abnormal state when the duration T of the moveless stateexceeds the reference time Tth. As a result, an abnormal state includinga microsleep state with eyes open can be determined, and an abnormalstate can be determined regardless of whether or not the occupant is ina state in which the eye opening degree D is not detected normally.

Further, the abnormal state determining unit 14 sets the reference timeTth to a value that differs depending on the autonomous driving level.As a result, the determination of an abnormal state on the basis of aproper reference time Tth can be implemented in accordance with theautonomous driving level of the vehicle 1.

Further, the image recognition processing includes the process ofdetecting the eye opening degree D of the occupant, and the abnormalstate determining unit 14 sets the reference time Tth to a value thatdiffers depending on the reliability of a result of the detection of theeye opening degree D. As a result, the determination of an abnormalstate on the basis of a proper reference time Tth can be implemented inaccordance with the reliability of the result of the detection of theeye opening degree D.

Further, the image recognition processing includes the process ofdetecting the eye opening degree D of the occupant, the determiningmethod used by the moveless state determining unit 13 includes a methodusing a result of the detection of the eye opening degree D and a methodnot using a result of the detection of the eye opening degree D, and,when the image recognition processing unit 12 has failed in thedetection of the eye opening degree D, the moveless state determiningunit 13 determines whether or not the occupant is in the moveless stateby using the method not using a result of the detection of the eyeopening degree D. As a result, the determination of the moveless stateusing a proper method can be implemented in accordance with the successor failure of the detection of the eye opening degree D.

Further, the image recognition processing includes the process ofdetecting the face area A1 in the captured image I, and, when either theamount of movement of the face area A1 or the amount of change in thesize of the face area A1 is less than the reference quantity, themoveless state determining unit 13 determines that the occupant is inthe moveless state. As a result, for example, the first, second, andthird concrete examples can be implemented.

Further, the captured image I includes a first image (I_(n)) and asecond image (I_(n+1)) which are captured at mutually different times bythe camera 2, the face area A1 includes a first face area (A1 _(n)) inthe first captured image (I_(n)) and a second face area (A1 _(n+1)) inthe second captured image (I_(n+1)), and the moveless state determiningunit 13 calculates the amount of movement on the basis of the ratio ofthe area of a part in which the first face area (A1 _(n)) and the secondface area (A1 _(n+1)) overlap each other to the area of either the firstface area (A1 _(n)) or the second face area (A1 _(n+1)). As a result,for example, the first concrete example can be implemented.

Further, the image recognition processing includes the process ofdetecting an inclination angle θ1 of an occupant's head, and, when theinclination angle θ1 is equal to or greater than the reference angle andeither the amount of movement or the amount of change is less than thereference quantity, the moveless state determining unit 13 determinesthat the occupant is in the moveless state. As a result, for example,the fourth and fifth concrete examples can be implemented.

Further, the image recognition processing includes the process ofdetecting the eye opening degree D of the occupant, and the movelessstate determining unit 13 determines whether or not the occupant is inthe moveless state on the basis of a change in the eye opening degree D.As a result, for example, the sixth and seventh concrete examples can beimplemented.

Further, the image recognition processing includes the process ofdetecting the face area A1 in the captured image I, and the movelessstate determining unit 13 generates a reference histogram indicating thepositions of the centers of the face areas (A1 ₀ to A1 _(n−1)) inmultiple images (I₀ to I_(n−1)) captured during a predetermined timeperiod and a histogram for comparison indicating the positions of thecenters of the face areas (A1 ₀ to A1 _(m−1)) in multiple images (I₀ toI_(m−1)) captured during another predetermined time period, anddetermines whether or not the occupant is in the moveless state bycomparing the reference histogram and the histogram for comparison. As aresult, for example, the eighth concrete example can be implemented.

Further, the image recognition processing includes the process ofdetecting the facial expression of an occupant, and the moveless statedetermining unit 13 determines whether or not the occupant is in themoveless state on the basis of presence or absence of a change in thefacial expression. As a result, for example, the ninth concrete examplecan be implemented.

Further, the image recognition processing includes the process ofdetecting the inclination angles θ2, θ3, and θ4 of an occupant'sshoulders, arms, and head, by using a skeleton model for the occupant,and the moveless state determining unit 13 determines whether or not theoccupant is in the moveless state on the basis of presence or absence ofa change in the inclination angles θ2, θ3, and θ4. As a result, forexample, the tenth concrete example can be implemented.

Further, the warning output control device 200 of Embodiment 1 includesthe occupant state determining device 100, and the warning outputcontrol unit 15 that performs control to output a warning when theabnormal state determining unit 14 determines that the occupant is in anabnormal state. As a result, when the driver's abnormal state occurs, awarning to that effect can be outputted.

Further, the warning output control unit 15 performs at least one ofcontrol to cause the display device 3 to display an image for warning,control to cause the sound output device 4 to output a sound forwarning, and control to cause the wireless communication device 5 totransmit a signal for warning. As a result, when the driver's abnormalstate occurs, a warning to that effect can be outputted to inside oroutside the vehicle.

Further, the occupant state determining method of Embodiment 1 includes:the step ST1 of, by the image data acquiring unit 11, acquiring imagedata indicating an image I captured by the camera 2 used for imagecapturing in the vehicle cabin; the step ST2 of, by the imagerecognition processing unit 12, performing image recognition processingon the captured image I by using the image data; the step ST3 of, by themoveless state determining unit 13, determining whether or not anoccupant is in the moveless state by using a result of the imagerecognition processing; and the step ST4 of, by the abnormal statedetermining unit 14, determining whether or not the occupant is in anabnormal state due to decrease in the degree of awakening by using aresult of the determination by the moveless state determining unit 13,and the abnormal state determining unit 14 determines that the occupantis in an abnormal state when the duration T of the moveless stateexceeds the reference time Tth. As a result, an abnormal state includinga microsleep state with eyes open can be determined, and an abnormalstate can be determined regardless of whether or not the occupant is ina state in which the eye opening degree D is not detected normally.

Embodiment 2

FIG. 12 is a block diagram showing a main part of a warning outputcontrol device according to Embodiment 2. Referring to FIG. 12, anoccupant state determining device 100 a and the warning output controldevice 200 a of Embodiment 2 will be explained. In FIG. 12, the sameblocks as those shown in FIG. 1 are denoted by the same reference signs,and an explanation of the blocks will be omitted hereafter.

An operation state information acquiring unit 16 acquires informationindicating a state of a driver's operation on a vehicle facility 6(referred to as “operation state information” hereafter). Concretely,for example, the operation state information acquiring unit 16 acquiresthe operation state information from the vehicle facility 6 via anin-vehicle network such as a not-illustrated controller area network(CAN).

The vehicle facility 6 is related mainly with the traveling of a vehicle1. The vehicle facility 6 includes, for example, the steering, theaccelerator pedal, the brake pedal, the turn signal, the doors, and theshift lever.

The operation state information acquiring unit 16 outputs the acquiredoperation state information to a moveless state determining unit 13 a.

The moveless state determining unit 13 a determines whether or not thedriver is in a moveless state by using a result of image recognitionprocessing by an image recognition processing unit 12, and the operationstate information outputted by the operation state information acquiringunit 16.

More specifically, the moveless state determining unit 13 a determinespresence or absence of the driver's operation on the vehicle facility 6by using the operation state information outputted by the operationstate information acquiring unit 16. Further, the moveless statedetermining unit 13 a determines presence or absence of the driver'smovement by means of at least one of methods of first to tenth concreteexamples, by using the result of the image recognition processing by theimage recognition processing unit 12. When determining that there is nodriver's operation on the vehicle facility 6 and there is no movement ofthe driver, the moveless state determining unit 13 a determines that thedriver is in the moveless state. Because the first to tenth concreteexamples are the same as those explained in Embodiment 1, an explanationof the concrete examples will not be repeated.

A main part of the occupant state determining device 100 a isconstituted by an image data acquiring unit 11, the image recognitionprocessing unit 12, the moveless state determining unit 13 a, anabnormal state determining unit 14, and the operation state informationacquiring unit 16. A main part of the warning output control device 200a is constituted by the occupant state determining device 100 a and awarning output control unit 15.

Further, because the hardware configuration of the main unit of thewarning output control device 200 a is the same as that explained byreference to FIG. 6 in Embodiment 1, an illustration and an explanationof the hardware configuration will be omitted hereafter. Morespecifically, the function of the moveless state determining unit 13 amay be implemented by either a processor 41 and a memory 42, or aprocessing circuit 43. Further, the function of the operation stateinformation acquiring unit 16 may be implemented by either the processor41 and the memory 42, or the processing circuit 43.

Next, the operation of the warning output control device 200 a will beexplained by referring to a flowchart of FIG. 13. The warning outputcontrol device 200 a repeatedly performs the processing shown in FIG. 13in a state in which, for example, the power is switched on (moreconcretely, in a state in which the ignition power source of the vehicle1 is switched on).

First, in step ST6, the operation state information acquiring unit 16acquires the information about the operation state of the vehiclefacility 6. The operation state information acquiring unit 16 outputsthe acquired operation state information to the moveless statedetermining unit 13 a.

Then, in step ST1, the image data acquiring unit 11 acquires image dataindicating a captured image I from a camera 2. The image data acquiringunit 11 outputs the acquired image data to the image recognitionprocessing unit 12.

Then, in step ST2, the image recognition processing unit 12 performs theimage recognition processing on the captured image I by using the imagedata outputted by the image data acquiring unit 11. Because a concreteexample of the image recognition processing is as explained inEmbodiment 1, an explanation of the concrete example will not berepeated.

Then, in step ST3 a, the moveless state determining unit 13 a determineswhether or not the driver is in the moveless state by using the resultof the image recognition processing by the image recognition processingunit 12, and the operation state information outputted by the operationstate information acquiring unit 16. Because a concrete example of adetermining method used by the moveless state determining unit 13 a isalready explained, an explanation of the concrete example will not berepeated.

When it is determined that the driver is in the moveless state (“YES” instep ST3 a), the abnormal state determining unit 14, in step ST4,determines whether or not the duration T of the moveless state exceeds areference time Tth. This duration T is calculated by the abnormal statedetermining unit 14.

When it is determined that the duration T of the moveless state exceedsthe reference time Tth (“YES” in step ST4), the warning output controlunit 15, in step ST5, performs control to output a warning. Because aconcrete example of a method of outputting a warning, the method beingused by the warning output control unit 15, is as explained inEmbodiment 1, an explanation of the concrete example will not berepeated.

By thus using the operation state information in addition to the resultof the image recognition processing, whether or not the driver is in themoveless state can be determined more correctly.

In a case in which an ECU disposed in the vehicle 1 has an on-boarddiagnostics (OBD) function, the operation state information acquiringunit 16 may acquire the operation state information outputted by the OBDfunction.

Further, in a case in which the moveless state determining unit 13 aimplements the eighth concrete example, the moveless state determiningunit 13 a may detect a time t1 when a calibration condition isestablished by using the operation state information. More specifically,the calibration condition may be one that the driver is not operatingthe vehicle facility 6. As a result, a reference histogram suitable forcomparison with a histogram for comparison (i.e., a reference histogrammaking it possible to determine whether or not the driver is in themoveless state more correctly by means of the comparison) can begenerated.

Further, as the occupant state determining device 100 a, the samevarious variants as those explained in Embodiment 1, i.e., the samevarious variants as those of the occupant state determining device 100can be employed.

Further, as the warning output control device 200 a, the same variousvariants as those explained in Embodiment 1, i.e., the same variousvariants as those of the warning output control device 200 can beemployed.

As mentioned above, the occupant state determining device 100 a ofEmbodiment 2 includes the operation state information acquiring unit 16that acquires operation state information indicating the state of anoccupant's operation on the vehicle facility 6, and the moveless statedetermining unit 13 a determines whether or not the occupant is in themoveless state by using the result of the image recognition processing,and the operation state information. By using the operation stateinformation in addition to the result of the image recognitionprocessing, whether or not the driver is in the moveless state can bedetermined more correctly.

It is to be understood that any combination of two or more of theabove-mentioned embodiments can be made, various changes can be made inany component according to any one of the above-mentioned embodiments,and any component according to any one of the above-mentionedembodiments can be omitted within the scope of the present disclosure.

INDUSTRIAL APPLICABILITY

The occupant state determining device, the warning output controldevice, and the occupant state determining method of the presentdisclosure can be applied to, for example, so-called “driver monitoringsystems.”

REFERENCE SIGNS LIST

1 vehicle, 2 camera, 3 display device, 4 sound output device, 5 wirelesscommunication device, 6 vehicle facility, 11 image data acquiring unit,12 image recognition processing unit, 13, 13 a moveless statedetermining unit, 14 abnormal state determining unit, 15 warning outputcontrol unit, 16 operation state information acquiring unit, 21 facearea detecting unit, 22 face feature point detecting unit, 23 eyeopening degree detecting unit, 24 inclination angle detecting unit, 25facial expression detecting unit, 31 human body area detecting unit, 32skeletal frame feature point detecting unit, 33 inclination angledetecting unit, 41 processor, 42 memory, 43 processing circuit, 100, 100a occupant state determining device, and 200, 200 a warning outputcontrol device.

The invention claimed is:
 1. An occupant state determining devicecomprising: processing circuitry to acquire image data indicating animage captured by a camera used for image capturing in a vehicle cabin;perform image recognition processing on the captured image by using theimage data; determine whether an occupant is in a moveless state byusing a result of the image recognition processing; and determinewhether the occupant is in an abnormal state due to decrease in a degreeof awakening by using a result of the determination, wherein the imagerecognition processing includes a process of detecting a face area inthe captured image, when an amount of movement of the face area or anamount of change in a size of the face area is less than a referencequantity, the processing circuitry determines that the occupant is inthe moveless state, and the processing circuitry determines that theoccupant is in an abnormal state when a duration of the moveless stateexceeds a reference time.
 2. The occupant state determining deviceaccording to claim 1, wherein the processing circuitry sets thereference time to a value that differs depending on an autonomousdriving level.
 3. The occupant state determining device according toclaim 1, wherein the image recognition processing includes a process ofdetecting an eye opening degree of the occupant, and the processingcircuitry sets the reference time to a value that differs depending onreliability of a result of the detection of the eye opening degree. 4.The occupant state determining device according to claim 1, wherein theimage recognition processing further includes a process of detecting aneye opening degree of the occupant, a determining method used by theprocessing circuitry includes a method using a result of the detectionof the eye opening degree and a method not using a result of thedetection of the eye opening degree, and when the processing circuitryhas failed in the detection of the eye opening degree, the processingcircuitry determines whether the occupant is in the moveless state byusing the method not using a result of the detection of the eye openingdegree.
 5. The occupant state determining device according to claim 1,wherein the captured image includes a first image and a second imagewhich are captured at mutually different timing by the camera, the facearea includes a first face area in the first captured image and a secondface area in the second captured image, and the processing circuitrycalculates the amount of movement on a basis of a ratio of an area of apart in which the first face area and the second face area overlap eachother to an area of either the first face area or the second face area.6. The occupant state determining device according to claim 1, whereinthe image recognition processing further includes a process of detectingan inclination angle of the occupant's head, and when the inclinationangle is equal to or greater than a reference angle and the amount ofmovement or the amount of change is less than the reference quantity,the processing circuitry further determines that the occupant is in themoveless state.
 7. The occupant state determining device according toclaim 1, wherein the image recognition processing further includes aprocess of detecting an eye opening degree of the occupant, and theprocessing circuitry further determines whether the occupant is in themoveless state on a basis of a change in the eye opening degree.
 8. Theoccupant state determining device according to claim 1, wherein theimage recognition processing further includes a process of detecting aface area in the captured image, and the processing circuitry furthergenerates a reference histogram indicating a position of a center of aface area in each of multiple images captured during a predeterminedtime period and a histogram for comparison indicating a position of acenter of a face area in each of multiple images captured during anotherpredetermined time period, and determines whether the occupant is in themoveless state by comparing the reference histogram and the histogramfor comparison.
 9. The occupant state determining device according toclaim 1, wherein the image recognition processing further includes aprocess of detecting a facial expression of the occupant, and theprocessing circuitry further determines whether the occupant is in themoveless state on a basis of presence or absence of a change in thefacial expression.
 10. The occupant state determining device accordingto claim 1, wherein the image recognition processing further includes aprocess of detecting inclination angles of the occupant's shoulder, arm,and head by using a skeleton model for the occupant, and the processingcircuitry further determines whether the occupant is in the movelessstate on a basis of presence or absence of a change in the inclinationangles.
 11. The occupant state determining device according to claim 1,wherein the processing circuitry further acquires operation stateinformation indicating a state of the occupant's operation on a vehiclefacility, wherein the processing circuitry determines whether theoccupant is in the moveless state by using the result of the imagerecognition processing and the operation state information.
 12. Awarning output control device comprising: the occupant state determiningdevice according to claim 1; and a warning output controller to performcontrol to output a warning when it is determined that the occupant isin an abnormal state.
 13. The warning output control device according toclaim 12, wherein the warning output controller performs at least one ofcontrol to cause a display device to display an image for warning,control to cause a sound output device to output a sound for warning,and control to cause a wireless communication device to transmit asignal for warning.
 14. An occupant state determining method comprising:acquiring image data indicating an image captured by a camera used forimage capturing in a vehicle cabin; performing image recognitionprocessing on the captured image by using the image data; determiningwhether an occupant is in a moveless state by using a result of theimage recognition processing; and determining whether the occupant is inan abnormal state due to decrease in a degree of awakening by using aresult of the determination, wherein the image recognition processingincludes a process of detecting a face area in the captured image, andwhen an amount of movement of the face area or an amount of change in asize of the face area is less than a reference quantity, it isdetermined, by the moveless state determining unit, that the occupant isin the moveless state, and determining that the occupant is in anabnormal state when a duration of the moveless state exceeds a referencetime.